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This paper introduces an uncertainty model being developed for the National Transonic 
Facility (NTF). The model uses a Monte Carlo technique to propagate standard 
uncertainties of measured values through the NTF data reduction equations to calculate the 
combined uncertainties of the key aerodynamic force and moment coefficients and 
freestream properties. The uncertainty propagation approach to assessing data variability is 
compared with ongoing data quality assessment activities at the NTF, notably check 
standard testing using statistical process control (SPC) techniques. It is shown that the two 
approaches are complementary and both are necessary tools for data quality assessment and 
improvement activities. The SPC approach is the final arbiter of variability in a facility. Its 
result encompasses variation due to people, processes, test equipment, and test article. The 
uncertainty propagation approach is limited mainly to the data reduction process. However, 
it is useful because it helps to assess the causes of variability seen in the data and 
consequently provides a basis for improvement. For example, it is shown that Mach number 
random uncertainty is dominated by static pressure variation over most of the dynamic 
pressure range tested. However, the random uncertainty in the drag coefficient is generally 
dominated by axial and normal force uncertainty with much less contribution from 
freestream conditions. 
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axial force coefficient 
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drag coefficient 
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chord Reynolds number 
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lift coefficient 
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normal force coefficient 
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standard uncertainty 

F n 

normal force 

U c 

combined standard uncertainty 

Fa 

axial force 

U = ku c 

expanded uncertainty 

M, 

freestream Mach number 
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molecular weight 
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coverage factor 
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a measured quantity 
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freestream dynamic pressure 
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result of a data reduction equation 
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static and total pressure 
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angle of attack 
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universal gas constant 
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static density 
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reference area 

Aug 

estimated within-group standard deviation 

LN2 

liquid nitrogen 

2cr 

expanded uncertainty with k — 2 
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I. Introduction 

C ompetitive and environmental concerns are driving unprecedented levels of product performance requirements 
at minimum development cost, and aerodynamic performance is no exception. Fuel economy is a relevant 
example. For a commercial aircraft program, fuel economy targets dictated either by customer or environmental 
constraints may drive the selection of an aggressive wing design that optimizes drag at cruise condition, but which 
carries some risk at an off-design condition, e.g., excessive trailing-edge boundary layer separation. The 
aerodynamicist needs high confidence in his predictions of in-flight aerodynamic performance at both conditions if 
his recommendations for the cruise condition carry with them performance risk at another condition. An automobile 
development program will routinely be under regulatory pressure to achieve a difficult design target for fuel 
economy, with consumer pressure not to compromise styling and cabin room, both while ensuring adequate engine 
cooling flow and safe handling characteristics. Flundreds of hours of wind tunnel time and even more computer 
time will be expended, from the first reduced-scale clay model to the final pre-production prototype, to achieve a 
series of small aerodynamic improvements. Taken alone, any one of these may account for only a few tenths of a 
mile per gallon. Taken together, they may provide a significant improvement but many of them add cost to the final 
product. Again, the aerodynamicist needs an accurate understanding of each modification’s influence on fuel 
economy in real-world driving conditions if his design recommendations are going to result in increasing the 
vehicle’s cost to manufacture. 

In the past, a more conservative approach would likely have been taken in both examples, because the lower 
confidence level in the available design data would not have been able to mitigate the performance risk of the 
aggressive wing design or justify the additional program cost for the aerodynamic enhancements to the automobile. 
The resulting compromises that in the past would have been acceptable from a commercial or regulatory standpoint 
would no longer be acceptable. In this setting - where performance requirements tighten but program budgets 
decrease - the need is evident for high-fidelity data with which to make the right design decisions. This has 
implications for aerodynamic development tools. Techniques that were previously used primarily for research or 
concept development must be transitioned faster into providing design direction; tools that previously provided 
design direction now need to give high-precision answers that closely resemble free-air conditions. And prototype or 
flight testing has to be minimized as much as possible to contain program cost and thus the end customer’s cost. 
This trend means, for example, that computational methods are being relied upon deeper into the product 
development process, and design decisions from experimental results - primarily wind tunnel tests - are being made 
on the basis of ever-smaller observations of performance increments. 

This product development environment requires a structured and ongoing internal commitment to maintaining 
data quality: methods to assess overall tunnel performance 1 along with analytical tools 2 * " 4 , processes, training, and 
commitment of resources (especially tunnel time) to diagnose and understand the causes of data variability at the 
subsystem level. Through these activities, it is possible to define and pursue opportunities for continuous 
improvement in both the precision (repeatability) and accuracy (fidelity relative to free-air conditions) of the 
simulations provided to the customer. 

The purpose of this paper is to report on additional progress at the National Transonic Facility (NTF) toward 
enhancing data quality by introducing and exercising a comprehensive numerical model to assess data uncertainty at 
the NTF. Section II places this paper in context relative to other continuous improvement activities at the NTF. In 
Section III, we provide a high-level review of methods for estimating variability, as a means of both introducing the 
model and placing it in context relative to previous work at NTF in this area. We describe the numerical model in 
detail Section IV, the benchmark tests used to populate it in Section V, and present some results in Section VI. 

II. Data Quality Assessment at the NTF 

The National Transonic Facility 5 is a high-Reynolds number transonic wind tunnel capable of varying Mach 
number, Reynolds number, and dynamic pressure independently by pressurizing the circuit and operating in 
cryogenic (nitrogen) or warm air mode. It has a unique potential to provide the high-fidelity design data required in 
the product development environment just described. On the other hand, a cursory review of the literature describing 
its conception and development, even up to the time it became operational 6 , suggests that it was intended primarily 
as a research facility. Further, the NTF was the first of its kind to be built at large scale, and remains today the 
largest cryogenic wind tunnel in the world. This means that part of its initial mission necessarily became how to 
operate and maintain a large cryogenic wind tunnel . 7 ' 8 Although the NTF remains an essential national asset for 
high Reynolds number research, activities are ongoing 1 9 10 to address the data quality features mentioned above that 
are now necessary for product development testing. 
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In addition to data quality concerns, productivity and reliability enhancements must also be addressed 
continually. In that regard, several initiatives have been completed recently or are currently underway to assess and 
improve the productivity and reliability of the NTF. An on-site LN2 plant was completed in 2008. It produces more 
than 1.5 times the amount of LN2 than could be provided previously and allows the tanks to be filled while the 
tunnel continues to run. A full upgrade of the NTF control and data systems is currently in progress. The control 
system is being replaced with a modern, programmable system. The new data system will provide a 5-fold increase 
in throughput, a 4-fold increase in the scan rate, and it will double the number of A/D channels, all of which will 
have 16-bit resolution. The calculation base will be converted entirely to a commercial off-the-shelf product with 
time-aligned input signals so that continuous-sweep data acquisition and reduction is possible. 

III. Estimating Data Variability 

Several previous investigations at the NTF have either focused on uncertainty analysis or included elements of it. 
This section provides an overview of several approaches to characterizing data variability with particular emphasis 
on past work at the NTF. It is neither exhaustive nor rigorous, but rather an attempt to explain and then compare, in 
simple terms, the relative strengths and weaknesses of the different approaches. 

A. Regression Statistical Analysis (RSA) 

Wahls et al. 11 investigated primarily short-term (within-test) variability on a commercial transport model. 
Mineck and Pendergraft 1 ’ used a similar analytical approach but with focus on test-to-test variability. They used 
classical multiple linear regression techniques to characterize spread in the results. Their approach was to use 
monomials as basis functions from one set of measured data (such as angle of attack) to develop regressions that 
provide the best fit to another measured variable (such as lift coefficient). In particular, they formed a single best- 
estimate curve fit from combined sets of identical inns. They referred to this approach as Regression Statistical 
Analysis (RSA). An example is shown in Fig. 1. A statistical estimate of variability is obtained with this approach 
by calculating either the confidence interval or prediction interval (which is shown in the figure). Both are functions 
of the standard error the fit, the local data density, and the ^-statistic for the desired confidence level. The confidence 
interval estimates the bounds within which the true mean is likely to fall, at the specified level of confidence. The 
prediction interval indicates the band within which a single future observation can be expected fall, at the specified 
level of confidence. The prediction interval seems more appropriate in most cases - in practice, the technique is used 
while the test is progressing to make sure repeat runs don’t show any surprises. 

While the RSA fit provides a metric to assess variability, it lacks somewhat in rigor. For example, unless 
variances (uncertainties) are specified for each dependent variable in the fit, there is no way to assess the quality of 
the model 1 ' itself (i.e., 4 th order polynomials in the case Fig. 1), so the choice of basis functions to use is arbitrary, 
although rules of thumb 11 and past experience certainly provide guidance. In addition, standard regression 
techniques assume no variability in the dependent variable, which is clearly not the case when a measured parameter 
is used. As a result. Fig. 1 shows that the prediction interval for the same independent variable ( C D ) differs by 
almost a factor of 2 when a is used for the dependent variable instead of Cl- In spite of these drawbacks, the RSA 
approach is a quick and visual method to obtain an estimate of variability and only requires data that are available as 
the test progresses. 

B. Statistical Process Control (SPC) 

A long-term initiative at the Langley Research Center, led by Hemsch 1 ' 4 14 and co-workers, has resulted in an 
established data quality assurance program at the NTF and other Langley wind tunnels. It is based on Shewhart 
control charts to monitor and maintain statistical process control (SPC) of the key aerodynamic force and moment 
coefficients. SPC techniques have a long and well-vetted history of application across many different disciplines 15 , 
most notably as a tool for quality control in manufacturing. For wind tunnel applications in particular, Hemsch et 
al. 14 provide an excellent introduction to the topic in general and describe the check standard programs they 
designed to balance economic and technical considerations. A subsequent paper 3 4 focused on the NTF specifically, 
showing initial results of the approach from three separate tests. 

Fig. 2 shows a process control chart for the axial force coefficient C A on the NTF’s Pathfinder I transport 
model 16 check standard. This chart contains data from 2004-2010. The top chart shows the group average and the 
bottom chart shows the group range. A ‘group’ in this case is the two measurements of C A at the common test 
conditions shown in the caption. Each measurement is one point in a pitch polar run, and between these two runs 
are back-to-back inverted pitch polar runs. Separate groups on the chart represent either a different test series (i.e., a 
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subsequent repeat of the pitch polar after a data set at different freestream conditions has been measured) or a 
different test (installation) altogether. 

In Fig. 2, the estimated in-group variability is about 2 cr wg = ±0.0001, or 1 count of drag for a transport model. A 
coverage factor of is 2 applied to a wg , which is traditionally considered to approximate a 95% confidence level. The 
group average data fall within the upper and lower control limits on the chart, so the process being measured may be 
said to be ‘in statistical control.’ Whether the resulting estimated variability is satisfactory is a different question, 
which is answered, of course, by considering the variability in relation to customer requirements. 

Establishing and maintaining a process control program is arguably the most important aspect of a wind tunnel’s 
data quality assurance program. The data are used to define the control limits, so the control chart rigorously defines 
the tunnel’s variability. It is not confined to repeatability (as is RSA) nor to the data reduction process (as is 
uncertainty propagation discussed below), but instead includes all sources of variability in the tunnel operation - 
people, processes, equipment, and test article. However, such programs are also expensive because regular tunnel 
entries are required; this is amplified for tunnels with a large operating envelope and a diversity of test articles. 
Even so, SPC programs are the only way to monitor the actual variability of the tunnel, and demonstrate this to the 
customer so that he can assign appropriate confidence to his own results when making product decisions. 

It is difficult with SPC techniques to drill down into the variability - to answer the question of why the 
variability exhibits the levels it does. The very rigor of the method requires that new tests be designed and new data 
acquired and analyzed to answer the question. The uncertainty propagation method described next helps to answer 
the question, but at the expense in some cases of replacing measured data and statistical rigor with engineering 
judgment. 

C. Uncertainty Propagation 

1. Definitions 

Up to this point, we have avoided as much as possible using terms like error, accuracy, and uncertainty, instead 
using (and conveniently neglecting to define) the term variability. Accuracy is the closeness of agreement between 
a measurement result and its true value. However, the true value is not known, nor can it be, so it follows that 
accuracy is only a qualitative term. Similarly, error is the degree of inaccuracy, i.e., the difference between the 
measured result and true value, so it too is unknowable and thus qualitative. Uncertainty is a quantified estimate of 
the error, hopefully based on a statistical analysis of measured data, but, when necessary for economic or other 
reasons, based on experience and judgment. These notions are based on the definitions given in the ISO Guide to 
the Expression of Uncertainty’ in Measurement' 1 . 

It is often helpful in understanding and analyzing error sources to classify them. A common approach is to 
classify errors as random or systematic, also known as precision and bias errors, respectively. The simple definitions 
for these terms are that a random error contributes to data scatter whereas a systematic error does not 18 . 

Systematic errors sources are usually related to residual error after calibration, data acquisition methods, operator 
interaction (deciding when the tunnel is ‘on point’), conceptual errors (measuring a distorted artifact of what was 
intended) and math models used to adjust data (e.g., blockage corrections). Random error is related to the scatter of 
successive measurements that are made under nominally identical conditions. It results from randomly occurring 
factors in the measurement system that cannot be identified and/or controlled. If such factors could be identified and 
controlled or eliminated, then the random error would be reduced. 

The classification of errors as random or systematic is context-specific and must be considered carefully in each 
situation. For example, a transducer output during calibration may be subject to random variations due to 
uncontrolled environmental changes in the laboratory. But that random error is fossilized 19 into the calibration’s gain 
coefficient, so it is a systematic error in the context of the wind tunnel’s use of the transducer. Conversely, test 
article misalignment is a fixed systematic error for any one test article installation, but viewed across installations it 
is a random error in the sense that it contributes to data scatter. The ISO Guide 17 recommends against the 
classification of errors as random or systematic because of ambiguities such as these, and also because there is 
fundamentally no difference in how the uncertainties are combined and used in subsequent uncertainty calculations. 
However, we retain the systematic and random terminology in this paper as a helpful tool to identify and classify 
error sources, even if the classifications sometimes contain ambiguities. 

2. Propagation Equation 

The method of uncertainty propagation estimates the error in a calculated measurement result by propagating 
estimated errors in the measured quantities through the data reduction equation that leads to that result. The 
uncertainty propagation equation is derived as a linearized Taylor series expansion about the true result. The true 
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result is then replaced by its estimate - the sum of the measured result and the error estimates (uncertainties). Ref. 
20 provides a complete derivation. The Guide 17 defines the standard uncertainty u as an estimate for the standard 
deviation of a measured quantity. The uncertainty propagation equation is used to determine the combined standard 
uncertainty u c of a result that is calculated from measured quantities. For a result y that is calculated as a function of 
N measured variables x h the combined standard uncertainty is the square root of the combined variance u c 2 : 
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The second term accounts for correlations between the measurements as, for example, in the case of two 
transducers that would inherit a common portion of their uncertainty if both had been calibrated previously against 
the same standard. 

As a simple example, the density of an ideal gas is a function of the pressure, temperature, the universal gas 
constant, and the molecular weight of the gas: 
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We will compare Eq. (3) to its equivalent formulation as a Monte Carlo simulation later in the paper. 

3. Coverage Factor 

A coverage factor k is usually applied to the standard uncertainty to obtain the expanded or overall uncertainty as 
U = ku c . The purpose of the coverage factor is to provide an interval about y that can be expected to encompass 
‘most’ measurement results likely to be realized, so that the final result is expressed as Y = y ± U. Equation (1) 
requires no assumptions about the underlying probability distributions of the errors in u(x). However, when k is 
chosen, it brings an implicit assumption about the nature of these distributions 20 . A value of k = 2 or 3 is common; if 
the errors in x t from Eq. (1) are normally distributed, k— 2 corresponds approximately to a 95% confidence level. In 
the remainder of the paper, where we refer to a “2cr uncertainty,” it should be taken to mean the expanded 
uncertainty with k = 2. 

D. Comparison of the Approaches to Estimating Variability 

The three approaches to quantifying data variability discussed above are not in competition; in fact, all three are 
important because of their unique advantages. The uncertainty propagation approach is the only practical way to 
drill into the data reduction equations to get a sense of what error sources are likely to dominate at different 
conditions in the operating envelope. This provides an analytical basis for justifying continuous improvement 
initiatives, and can provide customers with a pre-test estimate of uncertainties they can expect for test conditions 
that are not close to those used in the regular benchmark entries. But it is limited to the data reduction process: it 
does not model how trip strips are applied and checked from one entry to the next, nor does it account for lunch 
breaks, shift changes, leaking valves, or loose cable connections. The SPC approach provides feedback on the 
overall facility performance (people, processes, and equipment) to both customers and operations staff. With its 
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firm statistical grounding, its answer over time — the ‘Voice of the Process’ — is the final arbiter in quantifying the 
variability of key facility parameters as well as whether the facility operation is ‘in control.’ It is expensive - it 
requires regular entries to sample the process - but it is critical to the success of a production test facility. Finally, 
the RSA approach is very quick and visual, and it requires only the test data at hand to get an estimate of the data 
variability. It is the least rigorous of the three approaches, but the reality is that any estimate of variability that can 
be had in near real-time is useful. 

IV. Data Reduction and Uncertainty Analysis Programs for the NTF 

The uncertainty model for the NTF is based on a Monte Carlo simulation technique, which is in turn wrapped 
around an offline NTF data reduction model. The uncertainty model propagates the standard uncertainties of 
measured data ( P h P s , etc.) through the NTF data reduction algorithm to generate combined standard uncertainties of 
the calculated parameters ( C D , C L , etc.). 

A. Offline Data Reduction 

The offline data reduction program for the NTF is a Matlab implementation that replicates most of the online 
data reduction process described in Ref. 21 along with several modifications made since then. The translation was 
originally conceived as a general tool to model the balance data reduction equations in several Langley wind 
tunnels. Subsequent work incorporated the real gas model and calculation of freestream properties so that 
aerodynamic force and moment coefficients could be calculated. When it was decided to extend the model’s use for 
uncertainty propagation, the various routines were vectorized as much as possible, which improved the data 
reduction time by about an order of magnitude. (A Monte Carlo simulation with 10,000 iterations takes a few 
seconds on a laptop PC.) The calculation starts with data that have already been averaged and converted to 
engineering units, but the model will eventually extend back to raw count data. 

B. Uncertainty Analysis Using Monte Carlo Numerical Simulation 

Coleman and Steele 20 describe the application of Monte Carlo simulation techniques applied to uncertainty 
analysis and provide several references to its use in developing or validating concepts such as assessing the quality 
of the coverage factor for small-sample analyses. The approach has also been applied previously in automotive wind 
tunnel and related applications by Walter and co-authors 4 ' 22 and by others in numerous applications outside of 
aerodynamics. 

A Monte Carlo simulation generates a synthetic population of test results. The data reduction algorithm is 
executed numerous (i.e., several hundred or thousand or more) times, with the measurands randomly perturbed on 
each iteration. The technique can use actual test data to represent the entire population of such measurements. The 
underlying assumption is that the errors do not vary rapidly as a function of the true result 13 , so a single physical 
realization can serve as a reasonable surrogate for the true result for the purpose of comparing it to synthetically- 
generated results. The simulation provides a synthetic population of ‘experimental’ results, whose standard deviation 
is taken as the combined standard uncertainty. 

As a simple example, the following pseudocode is the equivalent to the ideal-gas density data reduction and 
uncertainty given in Eqs. (2) and (3): 

% Experimental data - surrogate for the 'true' result 

T = 15 t 273.15; P = 101325; R = 8314.47215; W = 28.9644; 

% standard uncertainties 

stdUncT = 0.5; stdUncP = 50; stdUncR = 0.9e-03; stdUncW = 0.01; 

% generate populations of std uncertainties for each 'realization' of the 'test' 

uncT = (stdUncT) * rand ( numlts ) ; uncP = (stdUncP) * rand ( numlts ) ; 

uncR = (stdUncR) * rand ( numlts ) ; uncW = (stdUncW) * rand ( numlts) ; 

For i = 1 To numlts % perturb the surrogate experimental data 

T_perturb = T + uncT(i); % and recalculate the result 

P_perturb = P + uncP(i); 

R_perturb = R + uncR(i); 

W_perturb = W + uncW(i); 

rho(i) = P_perturb / T_perturb / (R_perturb/W_perturb ) ; 

Next i 

combinedStdUnc = stdev ( rho ); 
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In this example, rand represents a function call that generates an array of randomly distributed numbers with 0 
mean and unit standard deviation. When multiplied by the appropriate standard uncertainty, the resulting population 
of random numbers has the proper scale and dimensional units of the standard uncertainty. 

The random number generation can have any probability distribution appropriate to the measurand in question, 
although a Gaussian generator is normally used and was used for all results presented in this paper. The example 
above does not distinguish between error classifications. However, random and systematic uncertainty distributions 
can be generated separately using their respective standard uncertainties as scaling factors. In that case, both 
uncertainties are added to the surrogate data point in the perturbation loop. If correlated uncertainties can be 
estimated, they are simulated by adding the same random number to the two correlated variables on each iteration 20 . 
A similar technique is used when time-series data are being simulated 4 . In that case, an inner loop is created to 
represent each time step. A random uncertainty is added to the surrogate experimental data on each iteration of the 
inner loop, but the systematic uncertainty is added to the data only on each iteration of the outer loop. This 
simulates the same fixed bias (e.g., calibration error) that would exist on all of the time-series points but allows for 
appropriate perturbations due to random fluctuations (e.g., line noise) on the individual time-series points. 

A Monte Carlo simulation does not rely on the linearizing approximations inherent in Eq. (1), nor does it require 
calculating the partial derivatives numerically. It is easier to implement in all but the simplest of cases. Even a 
deceptively simple data reduction equation can lead to difficulties deriving the standard uncertainty. Consider: 

F F 

C D = — — since ^ — —cosa (4) 

qjs qj 


This equation for the drag coefficient appears as simple as Eq. (2) but in reality, the dynamic pressure is a function 
of the total and static pressure measurements, so implicit partial differentiation of q , with respect to P, and P s is 
required when applying Eq. (1) to Eq. (4). The resulting algebraic expressions become very lengthy, as Ref. 18 
shows for this specific example. In comparison, if the data reduction computer code already exists offline for Eq. (4) 
- with a subroutine to calculate q x given P, and P s - it is simple in modern programming languages to wrap a Monte 
Carlo algorithm around it. 

Another important advantage to Monte Carlo simulation techniques is for applications in which the data 
reduction equations do not exist in closed form and a closed-form approximation either is not possible or it fails to 
faithfully represent elements of interest in the uncertainty analysis. In the case of the NTF, a real gas equation of 
state is used, so the thermodynamic properties have to be calculated iteratively before q-, can be calculated and used 
in Eq. (4). A closed-form propagation equation for C D uncertainty is therefore not possible without simplifying 
approximations. Other examples where a Monte Carlo approach is necessary include algorithms with branching 
logic, built-in regressions, assessing sensitivity to moving average and other digital filter settings, and evaluating 
data rejection algorithms 4 ' 22 . 

C. Standard Uncertainties in the NTF Model 

The NTF uncertainty model is programmed generically to propagate both random and systematic standard 
uncertainties for most of the data that are measured in a test. This includes the six balance forces and moments, the 
three basic thermodynamic measurements (total and static pressure and total temperature), the model attitude (pitch, 
roll, and yaw angles), weight tares, and so on. In the results presented later, only the most salient of these have been 
exercised. The available systematic error estimates are described in this section, but they have not been exercised in 
any of the results presented later. The random error estimates are based on test data; the methods used to estimate 
these are deferred to Section V after the test program has been described. 

Although primary focus of the uncertainty model has been on estimating and propagating random errors, the 
current version of the model does simulate some of the more important systematic error sources. The NTF Flow 
Reference System (FRS) attempts to minimize uncertainty in P t and P, by providing the total and static pressures to 
the data system through combinations of absolute and differential measurements that depend on the magnitude of 
the differential pressure P t - P s 23 . A detailed model of the FRS has been incorporated into the uncertainty model 
using the vendor’s stated elemental systematic uncertainties and an unpublished analysis 24 of recent as-found 
calibrations of the system. The model includes a correlated systematic uncertainty based on the assumption that the 
transducers are calibrated by the same standard. Because plans are in place to replace the FRS, a generic uncertainty 
model and a model based on another vendor’s uncertainty statements are also incorporated in the model. These have 
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allowed a thorough analysis of the systematic uncertainty in Mach number, dynamic pressure, and Reynolds 
number, which will be incorporated into the specification for the new system. 

Detailed calibration data sheets for NTF balances currently do not separate errors due to repeat loadings, curve 
fit residual errors, inherited standard uncertainty, etc., as suggested in the Ref. 25 recommended practice. This 
precludes a contribution analysis to determine the error sources that dominate the balance calibration uncertainty. 
However, the overall uncertainty statements for each component have been incorporated into the model. 

Blockage corrections are available in the NTF, but they have not yet been incorporated into the offline data 
reduction code, and so are not available in the uncertainty model. However, Walker 10 26 has investigated the validity 
of the wall interference correction method used at NTF. In the process, he developed one of the very few rigorous 
uncertainty assessments ever to be undertaken for wall corrections. It should be possible without significant effort 
to incorporate those results into the model in the future. 

V, Benchmark Tests to Establish Random Uncertainties 

The uncertainty model development is part of a larger, ongoing effort to understand the sources of variability in 
the NTF and systematically address them as time and resources allow. As an initial part of that effort, the NTF’s 
conventional check standard test was extended in several ways, as described next. It was then possible to use the 
data from these tests to estimate random uncertainties for the measured variables directly from these test data. 

A. Test Program 

Although the NTF is primarily a high Reynolds number test facility, it can and frequently does function as a 
standard transonic pressure tunnel; customers often require Reynolds number data from the NTF to compare to 
testing done at other facilities around the world. Results in both air and cryogenic modes are therefore of interest. 
The NTF check standard is the Pathfinder I transport model 16 . For the tests reported here, the NTF 113B balance 
was used. The Pathfinder 1 is representative of a subsonic, energy-efficient transport with a wide-body fuselage. 

Table 1 shows the set of tunnel conditions selected 
for the tests. These conditions are representative of a 
large portion of stability and performance testing done at 
the NTF. They include low-speed conditions seen during 
high-lift configuration testing as well as high-speed 
conditions typical of commercial transport performance 
testing. The conditions shown are generally similar to 
the check standard testing used to generate the NTF 
process control charts such as Fig. 2. Check standard 
tests at NTF have historically been done only in air 
mode due to the high cost of cryogenic testing. In this 
case, cryogenic runs were also included. Mach number 
and dynamic pressure were held generally the same for 
these runs as for the air-mode runs, while using the 
cryogenic capability to elevate Reynolds number. (The 
Reynolds number in Table 1 is based on the mean chord 
length of the Pathfinder I, which is 5.74 in.) This 
ensured similar aero-elastic performance while also 
allowing a direct comparison of variability to the 
previous check standard tests. 

Run sets comprised four back-to-back pitch polars at each test condition. Each polar consisted of pitch-pause 
data points taken between -2 deg and +2 deg of pitch, in increments of 0.5 deg. All polars pitched from negative to 
positive angle of attack, but model roll alternated on each run, resulting in the following sequence of runs: upright, 
inverted, inverted, upright. This first and last of these constitutes a group in the SPC sense and is the same group 
definition currently used for NTF check standard tests (Fig. 2). The coupled upright and inverted runs were then 
used to calculate tunnel flow angularity in the normal way by comparing differences in Cat. Wind-off zeros were 
recorded at the beginning and end of each test condition to remove bias due to temperature drift in either the tunnel 
or control room. 

Each polar took approximately 3 minutes to complete resulting in about 20 minutes at each condition in a series. 
The total time to complete a series was about 3.5 hours, including wind-off zeros, pressure system calibrations, and 
changing tunnel total pressure. Five air series were collected over a period of approximately 3 days during NTF 


Table 1. Tunnel Test Conditions 


M m 

P„ psia 

T h ° F 

Re c x 10 6 

^oo? psf 

Air Mode 

0.800 

21.5 

120 

2.64 

910 

0.800 

39.9 


4.90 

1691 

0.189 

50 


1.87 

176 

0.289 

50 


2.80 

398 

0.500 

113.5 


10.30 

2424 

Cry’ogenic Mode 

0.800 

21.5 

-250 

11.00 

910 

0.800 

39.9 


20.50 

1691 

0.189 

50.0 


7.60 

176 

0.289 

50.0 


11.40 

398 

0.820 

59.0 


30.53 

2562 
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Test 193 in July 2008. During NTF Test 194 in September 2008, one air series was repeated followed by two 
cryogenic series. The latter series were run back-to-back with approximately 3 hours separating comparable runs. 

Numerous auxiliary runs were made in both tests for various diagnostic purposes related to the effort at NTF to 
understand sources of variability. Most of these are outside the scope of this paper, but some were of use in 
assessing random uncertainties of measured data. For example, 60 seconds of continuous data were acquired at 
many of the conditions in Table 1 for -2, 0, and 2 deg pitch. The results could then be re-reduced to simulate 
different averaging times. These runs resulted in true back-to-back measurements that could be compared to the 
within-group measurements of comparable points in the two upright runs of a group. 

B. Estimating Random Uncertainties at the Measurement System Level 

Random uncertainties for measured variables are derived in some sense from observations 20 - repeat 
measurements, auxiliary tests over a representative time frame, previous test experience, etc. The benchmark test 
data have been used as described in the following subsections to derive random uncertainties at the measurement 
system level rather than the elemental level 18 . This means, for example, that the random uncertainty in total pressure 
is estimated by analyzing repeated total pressure measurements themselves, rather than attempting to estimate and 
combine elemental sources such as unsteady flow and freestream turbulence effects, orifice and transducer 
environment, power supply fluctuations, etc. Analysis at the measurement system level includes all such sources but 
cannot distinguish between them. 

As noted above, five air series tests were conducted in NTF Test 193 (T193). For any one tunnel condition and 
model attitude (pitch and roll angles), this resulted in 10 nominally identical observations, or 20 such observations if 
no distinction is made between upright and inverted runs. For the two cryogenic series tests, there were only 8 
nominally identical observations, even when upright and inverted runs are considered together. Flowever, 
considerably more observations than this were used by accounting for or averaging through the influence of model 
attitude, as discussed below. 

1. Freestream Measurements 

The measurement systems that produce the calculated freestream Mach number, dynamic pressure, and Reynolds 
number are, of course, the total and static pressure, and total temperature. Thus, random uncertainties in the 
freestream conditions can be evaluated by propagating the random uncertainties in these measurements through the 
NTF data reduction equations. Initial evaluation of the variability in the freestream measurements showed a clear 
dependence on freestream dynamic pressure, but the scatter did not depend on the model attitude. Based on this 
initial analysis, the standard uncertainties for P t , P s , and T, were simply calculated as the standard deviation of these 
measurements over the entire set of data at a particular total pressure and temperature setpoint. With 9 pitch angles 
per run, 4 repeat runs per series (2 upright, 2 inverted) at each dynamic pressure, and 5 series repeats for air mode, 
this results in 180 available observations. For cryogenic mode, 72 observations were available. In a few cases, the 
setpoint total pressure or temperature were not quite identical to the other runs. In those cases, the entire run was not 
discarded from the calculation, rather than attempting to normalize all of the data sets, because there were still 
plenty of observations available to calculate the standard deviations. 

Total temperature showed no trend with dynamic pressure, so the average standard deviation over all dynamic 
pressure setpoints was used to calculate the random uncertainty. For total and static pressure, there was a clear 
correlation in most cases with the dynamic pressure. The uncertainties (with coverage factor 2) and regressions used 
in the Monte Carlo simulation are shown in Fig. 3, plotted against dynamic pressure. The trends in tunnel conditions 
are qualitatively typical of what was described by Hemsch et al. 9 , although they were referring to balance 
repeatability: at lower dynamic pressures, the uncertainty remains about the same, but after some threshold value, it 
begins to increase with dynamic pressure. For total and static pressure in cryogenic mode, the threshold appears to 
be around q x = 400 psf. It is much higher for total pressure in air, as shown in the plot. 

The plots in Fig. 3 show two sets of data along with the regression curves. Only the main data set (diamond 
symbols) were used to generate the regressions. The second data set consists of 36 back-to-back points, which were 
taken from the 60-second continuous data samples described above. The 60, 1 -second samples were acquired at -2, 
0, and +2 deg of model pitch, resulting in 180 sequential samples. These were re-calculated as 5-s averages at each 
pitch angle, for a total of 36 points. The resulting standard deviation of these 36 points (with coverage factor 2) is 
shown in the plots for comparison to the main data sets, which were obviously acquired over a longer time frame 
(hours and days compared to 3 minutes). It is interesting that the trends between variability in the two data sets are 
similar - within-run repeatability and between-group repeatability are comparable in magnitude by this measure. 

The final points of interest to note in Fig. 3 are that the magnitudes of the uncertainty in total pressure are about 
the same between cryogenic and air mode operation, whereas the static pressure uncertainty in cryogenic mode is 
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significantly higher than in air mode. Also, the static pressure uncertainties in both modes are significantly higher 
than the total pressure uncertainties. 

2. Balance Measurements 

Slight differences in flow angularity between runs result in a different angle of attack, which in turn produces 
wide variations in F A and F N even when C D or C L vary much less. As some of the variation in F A and F N is of course 
physically meaningful (and certainly not random), it is inappropriate in the case of balance measurements to 
consider points as ‘nominally identical’ between runs even when they have common pitch angle and tunnel 
condition setpoints. However, it is more reasonable to assume that flow angularity does not change over the 
relatively short time span between the first and last upright run at a particular dynamic pressure setpoint. This was 
verified by calculating flow angularity from both the first pair and second pair of upright/inverted inns. The 
differences were always small between these two but not necessarily between 111 ns or series. If flow angularity 
dependence is considered negligible over such short time spans, a simplistic residual analysis allows the variability 
in balance measurements to be compared between runs. For a particular tunnel condition and pitch angle setpoint in 
a series, the following differences were calculated: 

AF ai =F ai -(F ai+ F A4 )/2 

AF A4 =F A4 -(F ai+ F A4 )/2 

In this equation, F A1 is the axial force in the first run of the sequence, and F A4 is the axial force in the fourth run 
(second upright run). The residuals are equal and opposite. They represent a crude estimate of the variability 
between the two runs. A similar pair of residuals was calculated for the inverted runs that occurred between these 
two upright runs. With the series repeat runs (5 for air, 2 for cryogenic), this resulted in 20 and 8 data points, 
respectively, for comparison at one dynamic pressure and pitch angle setpoint. 

The standard deviations of these residuals were calculated and compared between pitch angle settings. Most of 
the resulting standard deviations showed no obvious trend with pitch. (In some cases, particularly lower dynamic 
pressures, the standard deviation was higher at the extreme pitch angle settings.) To avoid a two-dimensional 
regression (balance uncertainty vs. dynamic pressure and pitch angle), the standard deviations were averaged across 
all 9 pitch angles to arrive at the standard uncertainties for the balance forces. Instead of averaging standard 
deviations from individual pitch angles, an alternative approach of calculating the standard deviation of all residuals 
(all pitch angles, all repeats) gave similar results. 

The random uncertainties calculated in this way are shown for axial and normal force in Fig. 4. As in Fig. 3, only 
the main data sets were used to calculate the uncertainties; the standard deviations (with coverage factor 2) of the 
back-to-back measurements are shown only for comparison. The back-to-back plots are from the same runs as in 
Fig. 3, except they are calculated from 1-second averages instead of 5-second averages. 

As in Fig. 3, a strong correlation between balance variation and dynamic pressure is obvious. In this case, 
however, there doesn’t appear to be a threshold below which the uncertainties are independent of dynamic pressure 
- or if it exists, it is a low value. The uncertainty in cryogenic mode is between 1.5 and 2 times higher than in air 
mode for both axial and normal force over most of the dynamic pressure range. Also shown for reference are the 
overall calibration (systematic) uncertainties for the NTF 113B balance. For the axial force, the observed random 
uncertainty falls below the calibration uncertainty at all dynamic pressures. The normal force random uncertainty 
exceeds the calibration uncertainty for about q , > 600 psf in both air and cryogenic modes. 

Strong correlation with dynamic pressure was also observed in the other forces and moments (not shown). The 
pitching moment and side force tended to increase the most with dynamic pressure. The yawing and rolling 
moments were correlated with dynamic pressure but did not increase nearly as much over the dynamic pressure 
range tested. 

The variation of the freestream properties during a single run was not always random. In several back-to-back 
run sets we looked at in detail, total pressure appeared to be randomly distributed during the run, but static pressure 
decreased consistently by about 2 psf from the beginning of the run to the end, and the Mach number increased 
consistently by about 0.001. Considering the correlation of both balance forces and tunnel properties with dynamic 
pressure, the question was asked whether their random uncertainties are also correlated. Plots of F A vs. P s from 
high-speed (100 Hz) auxiliary data showed a ‘shotgun blast’ distribution. Figure 5 supports the lack of correlation. 
Using the high-speed data set, the cross-correlation coefficient function 27 was calculated for axial force and static 
pressure. Perfect correlation would be indicated by values of ±1, which is not the case in Fig. 5. A similar lack of 
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correlation was observed between other balance components and the static and total pressures. The in-run trends 
observed may instead be related to slight changes in wall interference during the pitch polar. 

3. Model Attitude and Flow Angularity 

The uncertainty in model pitch angle was estimated with the aid of auxiliary high-speed (100 Hz) time traces of 
the pitch angle feedback, which had been low-pass fdtered at 0.4 Hz. At each setpoint, the standard deviation was 
calculated from the final 2 seconds of data prior to a setpoint change. As there was no obvious trend with pitch 
angle setpoint, the standard uncertainty at a particular dynamic pressure was taken as the average of the 9 pitch 
angle standard deviations. Because similar high-speed roll angle data were not available, the roll angle uncertainty 
was assumed to be the same as pitch angle uncertainty. The pitch angle random uncertainty is shown in Fig. 6 for air 
and cryogenic modes. Correlation with dynamic pressure is obvious, but it is strong only in cryogenic mode. 

The random uncertainty in flow angularity was calculated using Hemsch’s 28 analysis for repeatability over a time 
frame comparable to this test: 


^ Flow Angle 0.633 deg 
= 0.00218 deg 

4. Closing Comments on Estimating Random Uncertainties 

The benchmark tests presented in this paper serve as a very good starting point for estimating random 
uncertainties in the NTF, but they are obviously limited to a single test article, with admittedly crude techniques 
used in some cases to arrive at the uncertainties. Going forward, future benchmark tests and customer test data can 
be used to refine these estimates and scale them 9 ' 14 28 for other balances and model sizes. In addition, the completed 
model requires frill implementation of the NTF systematic uncertainties so that in addition to repeatability and 
reproducibility, they can also properly compare their results to other simulations (i.e., other wind tunnels, prototype 
tests, and numerical tests). 


q , „ <290psf 
q m > 290 psf 


VI. Results 

The random uncertainties described above for the six balance components, three freestream measurement 
properties, model attitude angles, and flow angularity, were propagated through the uncertainty model using the 
appropriate data from representative runs in Table 1. For brevity, results in this section are confined mainly to M, 
and C D . 


A. Random Uncertainty Propagation in the Freestream Properties 

Figures 7 and 8 compare the calculated uncertainties in Mach number and dynamic pressure to the direct 
observations from the test data. As with the pressure data, the observed variations used data from all runs and model 
attitudes, under the assumption that the latter do not influence freestream properties. The agreement between the 
calculations and observations is reasonable in most cases, especially for dynamic pressure uncertainty, but the 
calculations are somewhat pessimistic at the lower Mach numbers in some cases. It is interesting that the 
uncertainty in Mach number increases with increasing dynamic pressure in cryogenic mode, but it decreases in air 
mode after q r: = 600 psf. We note from Table 1, however, that in the cryogenic case (Fig. 7b), the Mach number and 
dynamic pressure increase together, so the highest uncertainty corresponds to both maximum dynamic pressure and 
Mach number. In the air mode case (Fig. 7a), the highest uncertainty occurs in at the middle two dynamic pressures, 
both of which correspond to the highest Mach number tested in air mode. 

Figure 9 shows the contributions of P,, P s , and T, to the Mach number random uncertainty. A contribution 
analysis is calculated by running the simulation with all uncertainty sources set to zero except the one(s) of interest. 
As shown in the figure, the static pressure uncertainty dominates the Mach number uncertainty at the higher 
dynamic pressures. Considering that the components add as root-sum-squares to arrive at the combined result, the 
total pressure uncertainty is completely negligible except at the lowest dynamic pressures. The results for dynamic 
pressure uncertainty (not shown) exhibit the same trends as Mach number. This is perhaps not surprising in view of 
the relative magnitudes of the static and total pressure uncertainties in Fig. 3. This result leads to the conclusion that 
any attempt to improve the Mach number variation would need to focus on understanding and minimizing the static 
pressure variation. 
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B. Random Uncertainty Propagation in the Axial Force and Drag Coefficients 

Figure 10 compares results from a variety of sources to quantify the variation in the axial drag coefficient. 
Shown are results from the uncertainty propagation simulation, the NTF benchmark tests, SPC results from Flemsch 
et al. 9 , sequential measurements and RSA calculations from the present work, and RSA calculations from Wahls et 
al." All of the data sets are from the Pathfinder I model except those of Wahls et al. Their results were for a 767 
model; the error bars denote the range of results they obtained from various model configuration changes during the 
test. 

Some clarification may be in order regarding the inclusion of both between-group and within-group variation in 
Fig. 10. Flemsch et al. 9 defined their group as three back-to-back upright runs. Their between-group variation was 
calculated on the basis of a series of runs made during a single tunnel entry and over a time frame similar to the time 
frames of the tests in the present work. The NTF check standard SPC charts define a group similarly - as two 
upright runs, one before and one after two intervening inverted runs. Flowever, the calculation of between-group 
variation for the NTF SPC charts includes not only runs made over short time frames (hours up to several days), but 
also over multiple installations. In attempting to assess variation over a comparable time frame, it seems to make 
more sense to use within-group variation from the NTF SPC charts for comparison with both the Monte Carlo 
simulation results and the between-group results of Flemsch et al. 

The agreement between the different methods in Fig. 10 is quite good, particularly between the NTF check 
standard results, the results of Flemsch et al., and the propagation results. Considering the confidence limits on the 
standard deviation 9 20 for a Normal distribution (that is to say, the uncertainty of the uncertainty), the SPC and 
uncertainty results are effectively the same in Fig. 10. Not surprisingly, the RSA results exhibit more inconsistency 
than the more rigorous methods. 

Figure 1 1 shows the random uncertainty in the drag coefficient with increasing dynamic pressure for both air and 
cryogenic modes. Only RSA data are available for comparison in this case as SPC tests have not historically been 
done in the NTF; recall as well that the results of Wahls et al." are from a different model. The trends in the 
uncertainty propagation are similar in both cases. For air, however, the uncertainty levels out and becomes 
independent of dynamic pressure, whereas it begins to increase in the cryogenic case. This is not particularly 
surprising in view of the results in Figs. 3, 4, and 6 where the rate of increase with dynamic pressure in most of the 
component uncertainties is much higher in cryogenic mode than air mode. 

Figure 12 shows the contribution from the various sources to the combined random uncertainty at selected pitch 
angle and dynamic pressure setpoints. Air mode results are on the left and cryogenic mode results are on the right. 
The relative contributions are generally not very different between the air and cryogenic modes, only the levels. This 
is comforting at some level, because it suggests that the behavior of the measurement systems and probably the 
tunnel and model, are not fundamentally different in the two modes of operation. In all cases, axial force 
uncertainty, not unexpectedly, is a dominant contributor, but it is not necessarily the only important one. At the 
lowest dynamic pressure in air mode (Fig. 12a), the uncertainty contribution from the freestream measurements is 
the same as the axial force contribution. At the highest cryogenic dynamic pressure (Fig. 12f), axial force, normal 
force, and pitch angle uncertainties all contribute equally at the highest angle of attack. Generally, however, only 
the normal force uncertainty is an equal and in some cases larger contributor than the axial force. Taken together, 
the results in Fig. 12 suggest that if an improvement in drag coefficient random uncertainty is sought, focusing on 
axial force uncertainty alone would not be sufficient. 

The uncertainties in balance forces, pitch angle, and even the freestream tunnel conditions all exhibit increasing 
magnitudes and generally strong correlation with increasing dynamic pressure and Reynolds number, which 
propagates into the calculated uncertainties for the freestream properties and the drag coefficient. This may point to 
an aeroacoustic or aeroelastic interaction with the model or model support system, although this is speculative on the 
basis of the data set that was acquired in the test program reported here. Flowever, vibration problems related to such 
phenomena have been observed previously in the NTF 29 ’ 30 and have been found to be magnified at high Reynolds 
number (cryogenic) conditions. 


VII. Concluding Remarks 

Unprecedented levels of product performance requirements are driving design decisions and product costs on the 
basis of ever-smaller observations of performance increments. This environment dictates the requirement for high- 
fidelity wind tunnel data, even from wind tunnels that were put into operation decades ago. The need for accurate 
and precise wind tunnel results in turn drives the need for ongoing improvement in wind tunnel data quality. Tools 
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and resources are needed that not only assess the data quality but also provide a technical basis to justify specific 
improvement initiatives. 

An uncertainty propagation model based on a Monte Carlo technique has been developed for the NTF. This 
approach provides a drill-down capability to identify the relative influence of the uncertainties in measured data 
(e.g., total pressure) on the uncertainties of the calculated data (e.g., Mach number and drag coefficient) provided to 
the customer. It was shown that the tool is complementary to the more fundamental but high-level information 
provided by SPC. The model has been populated with initial estimates of random errors for the most salient 
measured parameters. The estimates are based on benchmark tests in both air and cryogenic mode, which were 
conducted as part of this project. 

Analysis of the benchmark test data showed that random uncertainties in the measured parameters - tunnel 
pressures, balance forces, and model pitch angle - were correlated with the tunnel’s dynamic pressure and increased 
with it. Further, the rate of increase was higher (and the correlation stronger) in cryogenic (high Reyolds number) 
mode than in air mode. The results of the uncertainty propagation showed that the uncertainties in both calculated 
Mach number and dynamic pressure are driven by the uncertainty in measured static pressure, while the uncertainty 
in drag coefficient is driven by both axial and normal force uncertainty. 

It was speculated that some of the increase in uncertainty as dynamic pressure and Reynolds number increase 
could be related to aeroacoustic or aeroelastic interactions with the model or model support system. Although the 
data set described in this paper was not sufficient to investigate this, vibration phenomena have been observed in 
previous studies and found to be magnified at higher Reynolds numbers. The solutions developed or suggested 
previously to mitigate vibration phenomena generally focused on safe operation and avoiding restrictions to the 
testing envelope. As improvements in data quality are sought due to new, customer-driven performance 
requirements, a return of focus to further refine previous solutions would seem an appropriate path. This a common 
approach in continuous improvement initiatives - a solution that was once sufficient must be revisited in view of 
new customer requirements. The metric of success for such activities in this case would shift from health monitoring 
to reducing data variability. As indicated in the results presented in this paper, reducing the random uncertainties in 
static pressure and in the normal and axial forces would have the largest impact on the Mach number and drag 
coefficient variability. 
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Figure 1. Example RSA results. 
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Figure 2. NTF Statistical Process Control chart for C A upright, q w = 1690 psf, M m = 0.80, a= 2 deg, air 
mode operation. 
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Figure 3. Random uncertainties for P, and P s estimated from measured data. 
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Figure 4. Random uncertainties estimated from measured balance data. 
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Figure 5. Cross-correlation coefficient function - static pressure and axial 
force ( M x = 0.8, q x = 1691 psf, cryogenic mode). 
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Figure 6. Random uncertainty in model pitch angle. 
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Figure 7. Calculated vs. observed random uncertainty in M x . 
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Figure 8. Calculated vs. observed random uncertainty in q^. 
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Figure 9. Random uncertainty drill-down for M x Note that component uncertainties add as root-sum- 
squares for the combined result. 
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Figure 10. Air-mode variability in C A . 
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-O- Monte Carlo Simulation (average over all AoA) 

A NTF SPC Charts Within-Group Variation (2005-2008), 2 deg AoA 
■ T137 Between -Group Variation (Hemsch et al., 2004) 

□ T193 - 1 min of continuous data reduced as 60 1-s averages 
X T193 RSA Between-Group 95% Prediction Interval -CAv. CN 4th order fit, dof = 31 
♦ RSA95% Prediction Interval -CAv. CN 4th order fit, dof= 34-38 (Wahls et al., 1995) 
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Figure 11. Random uncertainties in C D . 
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c) M x = 0.80, < 7 oo = 910 psf- air mode 
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d) M x = 0.80, q w = 910 psf- cryogenic mode 
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e) M m = 0.50, q a o = 2424 psf- air mode 


f) M m = 0.82, q x = 2562 psf- cryogenic mode 


Figure 12. Uncertainty drilldown for C D . Note that component uncertainties add as root-sum-squares for the 
combined result. 
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